Skip to content

feat: add llm-cli-gateway as optional multi-LLM orchestration add-in#866

Open
mr-k-man wants to merge 1 commit intogarrytan:mainfrom
mr-k-man:feat/llm-gateway-add-in-clean
Open

feat: add llm-cli-gateway as optional multi-LLM orchestration add-in#866
mr-k-man wants to merge 1 commit intogarrytan:mainfrom
mr-k-man:feat/llm-gateway-add-in-clean

Conversation

@mr-k-man
Copy link
Copy Markdown
Contributor

@mr-k-man mr-k-man commented Apr 6, 2026

Summary

  • Adds llm-cli-gateway as an optional gstack add-in via the contrib/add-tool/ pattern
  • When installed, 6 skills gain a "Multi-LLM Orchestration" section with contextual MCP tool recommendations
  • Adds Gemini as a third review voice, async parallel orchestration, and session continuity — complementary to existing CODEX_SECOND_OPINION / ADVERSARIAL_STEP patterns

What's included

Component Files
Contrib structure contrib/add-tool/README.md, contrib/add-tool/llm-gateway/* (README, tools.json, detection.sh, install.sh, uninstall.sh)
Resolver scripts/resolvers/llm-gateway.ts — reads tools.json, emits conditional markdown per skill with host-aware CLI filtering
Preamble detection 12 lines of bash in preamble.ts — detects llm-cli-gateway binary + per-CLI availability (claude, codex, gemini)
Template integration {{LLM_GATEWAY_CONTEXT}} added to review, investigate, plan-eng-review, plan-ceo-review, ship, retro
Tests 109 new tests (resolver, schema validation, host suppression, integration)

Design decisions

  • MCP server name: llm-cli-gw (avoids collision with simonw's llm tool)
  • Complementary: Does NOT replace existing Codex exec patterns — adds Gemini + async + sessions alongside them
  • Host filtering: Generalized ctx.host === t.requires_cli prevents self-invocation on any host
  • Async only for parallel work: review/ship use _async variants; investigate/plan/retro use sync
  • Graceful degradation: Resolver returns empty string if gateway not installed or tools.json missing

Test plan

  • 109 new tests pass (resolver unit, tools.json schema, host suppression, generated SKILL.md content, preamble detection)
  • 335 existing gen-skill-docs tests pass
  • 71 existing host-config tests pass (golden files updated)
  • bun run gen:skill-docs --host all succeeds for all 8 hosts
  • Manual: install llm-cli-gateway, run /review on a real diff, verify Multi-LLM section appears and tools work

Add llm-cli-gateway integration via the contrib/add-tool/ pattern. When
installed, 6 gstack skills gain a "Multi-LLM Orchestration" section with
contextual MCP tool recommendations for Gemini + Codex parallel reviews,
async job orchestration, and session continuity.

Components:
- contrib/add-tool/llm-gateway/: routing contract, detection, install/uninstall
- scripts/resolvers/llm-gateway.ts: resolver with host-aware CLI filtering
- Preamble detection for llm-cli-gateway binary + per-CLI availability
- {{LLM_GATEWAY_CONTEXT}} in review, investigate, plan-eng-review,
  plan-ceo-review, ship, retro templates
- 109 new tests (resolver, schema, integration, preamble)

Complementary to existing CODEX_SECOND_OPINION — does not replace it.
MCP server name: llm-cli-gw (avoids collision with simonw's llm tool).
@mr-k-man mr-k-man force-pushed the feat/llm-gateway-add-in-clean branch from 7a0c475 to 6d3eddd Compare April 6, 2026 20:30
@garagon
Copy link
Copy Markdown
Contributor

garagon commented Apr 6, 2026

@garrytan Second PR today with the same pattern as #862. Another external dependency from the same GitHub org wired into all 30+ generated SKILL.md files. 50 files touched, 2097 lines added.

Combined with #862 that is 102 files and 5400 lines of additions in a single day, both embedding external MCP servers into every gstack skill. The security and performance concerns from my review on #862 apply here as well.

Recommend not merging either PR. Both wire external MCP servers with no content sanitization into an LLM context that has shell access on every user's machine.

@mr-k-man
Copy link
Copy Markdown
Contributor Author

mr-k-man commented Apr 6, 2026

Thanks for the review @garagon. Want to clarify how this differs from #862 since the concerns don't apply the same way:

Not the same pattern. This PR has zero ReadMcpResourceTool calls, zero sqry:// URIs, and zero content loaded from external MCP servers into the LLM context. The resolver emits static markdown with MCP tool names — it tells the agent "here are tools you can call," not "read this external content."

Actual diff: 20 files, 1502 lines (not 50/2097 — the generated SKILL.md files were removed from the diff in the force-push before your comment).

Point-by-point on the #862 concerns:

#862 concern This PR (#866)
curl-pipe-bash install npm install -g or cargo install — no curl-pipe-bash
MCP resource content → LLM context Zero MCP resource reads. Only static tool name references
3 subprocesses per skill load 4x command -v only (~5ms total, POSIX, no subprocesses)
No content sanitization No external content enters the LLM context — nothing to sanitize
Generated SKILL.md in diff Removed — 20 files now

The resolver output is a conditional markdown block like:

If preamble shows LLM_GATEWAY: unavailable: skip this section entirely.
...
- `mcp__llm-cli-gw__gemini_request_async` — dispatch Gemini review in parallel

No external content is fetched, read, or injected. The MCP tools are called by the agent at runtime through the standard MCP protocol with normal tool-call semantics — same as any other registered MCP server.

Happy to discuss further or make changes if there are specific concerns with this approach.

@garagon
Copy link
Copy Markdown
Contributor

garagon commented Apr 6, 2026

To clarify: "same pattern" refers to the contribution pattern, not the resolver implementation. Two PRs in a single day from the same author wiring two separate dependencies from the same GitHub org across multiple gstack skills. The org was created recently.

Reviewed the llm-cli-gateway source code (https://github.com/verivus-oss/llm-cli-gateway). Some observations on what users would be installing:

  • The package hardcodes sqry as a default MCP server when no servers are specified (src/index.ts line 392). The two tools are designed to work together.
  • A flight recorder logs all user prompts and AI responses to a plaintext SQLite database at ~/.llm-cli-gateway/logs.db, enabled by default (src/flight-recorder.ts). There is no clear reason for a CLI gateway to silently record every conversation to disk without explicit user consent.
  • Spawned processes inherit the full process.env including all API keys and credentials in the user's environment (src/executor.ts line 165).
  • The package includes a dangerouslySkipPermissions flag that passes --permission-mode bypassPermissions to the invoked CLIs (src/index.ts line 731).
  • The npm package was published 6 days ago (https://www.npmjs.com/package/llm-cli-gateway).

The supply chain risk here is real. A lot of people use gstack daily and trust what gets merged. That is for @garrytan to decide.

Gus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants